Sports simulation system

ABSTRACT

A sports simulation system ( 100 ) comprises at least two imaging devices ( 128 ) capturing images of a projectile tracking region disposed in front of a display surface ( 124 ) from different vantages to detect a launched projectile traveling through the projectile tracking region towards the display surface; a projectile spin sensing unit ( 105 ) capturing images of a region at least partially overlapping with the projectile tracking region, each captured image comprising a projectile trail representing a travel path of the projectile when a projectile is present in the region during image capture; and at least one processing stage ( 104 ) receiving data from the imaging devices ( 128 ) and the projectile spin sensing unit ( 105 ) and determining the three-dimensional positions, velocity, acceleration and spin of a detected launched projectile traveling through the projectile tracking region, the three-dimensional positions, velocity, acceleration and spin being used by the at least one processing stage to calculate a trajectory of the launched projectile into a presented sports scene.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.13/123,321, entitled SPORTS SIMULATION SYSTEM, filed Oct. 7, 2009, whichis a U.S. National Phase patent application based on InternationalApplication Serial No. PCT/CA2009/001424 filed Oct. 7, 2009, entitled“Sports Simulation System,” which is based on U.S. Provisional PatentApplication Ser. No. 61/103,790 filed on Oct. 8, 2008, the disclosuresof which are incorporated herein by reference.

This application is related to U.S. patent application Ser. No.10/629,945 filed on Jul. 30, 2003 for an invention entitled “SportsSimulation System”, to U.S. patent application Ser. No. 11/195,017 filedon Aug. 2, 2005 for an invention entitled “Sports Simulation System” andto U.S. patent application Ser. No. 11/394,004 filed on Mar. 30, 2006for an invention entitled “Sports Simulation System”, the disclosures ofwhich are incorporated herein by reference.

FIELD OF THE INVENTION

The present invention relates generally to entertainment systems and inparticular to a sports simulation system.

BACKGROUND OF THE INVENTION

Sports simulation systems designed to simulate sports experiences arewell known in the art. In many conventional sports simulation systems, aplayer propels a sports projectile such as a ball, puck, arrow, dart,etc. at a target image presented on a display screen. The motion of thesports projectile is detected and imaged and an extrapolation of thetrajectory of the sports projectile is made. The extrapolated trajectoryis then used to determine a sports result. The displayed image is inturn updated to reflect the sports result thereby to provide the playerwith visual feedback and simulate a sports experience.

The goal of all sports simulation systems is to provide the player witha realistic sports experience. As a result, many variations of sportssimulation systems have been considered in attempts to simulateaccurately “real-life” sports experiences. For example, U.S. Pat. No.5,333,874 to Arnold et al. discloses a sports simulator having a housingand two arrays of infrared (IR) receivers and emitters positioned in thehousing. A launch area is established near one end of the housing. Auser can launch an object such as a golf ball located in the launch areaand drive the golf ball into the housing through the planes defined bythe arrays of IR emitters and against a screen positioned at one end ofthe housing. A computer is connected to the IR receivers, which detectthe passage of the object through the respective planes. Based upon thesignals from the IR receivers, the computer uses triangulationtechniques to determine the horizontal and vertical position, as well asthe velocity of the golf ball. The computer can also determine the spinof the golf ball and cause an image of the golf ball as it would haveappeared traveling away from the golfer had it not encountered thescreen to be displayed on the screen.

U.S. Pat. No. 5,443,260 to Stewart et al. discloses a baseball trainingand amusement apparatus that detects the speed and projected flight of abatted baseball. The apparatus includes a ball delivery device, a pairof detection planes, a computer and a video and simulation monitor. Thedetection planes are parallel to one another and are spaced apart by adistance such that a batted ball passing through the detection planeswould be a fair ball in a real baseball game. Each detection planeincludes a rigid frame that supports a pair of optical scanners and apair of light sources. The optical scanners and light sources arepositioned at opposite top corners of the rigid frame and are aimeddownwardly into the region encompassed by the frame.

During use, the ball delivery apparatus delivers a baseball towards aplayer positioned in front of the detection planes. When the playerstrikes the baseball with a bat and the baseball travels through thedetection planes, the optical scanners capture images of the baseball.The images are processed to determine the coordinates of the baseball asit passes through each of the detection planes as well as the velocityof the baseball. A simulated trajectory of the baseball is thencalculated using the determined coordinate and velocity information. Thesimulated trajectory information is used to update the graphical imagespresented on the monitor so that the simulated flight of the battedbaseball is displayed to the player thereby to simulate a battingexperience.

U.S. Pat. No. 5,649,706 to Treat, Jr. et al. discloses a huntingsimulator for in-flight detection of a launched missile such as anarrow. The hunting simulator includes a screen and a projector forprojecting a moving target on the screen. Electromagnetic radiationemitters are positioned in front of the screen adjacent its opposite topcorners and illuminate a plane in front of the screen. Sensors are alsopositioned adjacent the opposite top corners of the screen and areresponsive to the electromagnetic radiation emitters. Retroreflectivetape extends along opposite sides of the plane.

During use, when an arrow is launched at the screen and passes throughthe plane, the sensors detect the presence of the arrow and generateoutput. The output of the sensors is used to determine the coordinatesof the arrow as well as the velocity of the arrow. A simulatedtrajectory of the arrow is then calculated and the graphical imagespresented on the screen are updated accordingly to reflect the flight ofthe launched arrow. In this manner, a hunting experience is simulated.

U.S. Pat. No. 5,768,151 to Lowy et al. discloses a system fordetermining the trajectory of an object in a sports simulator. Thesystem includes a baseball throwing device to deliver a baseball towardsa player area. A projector adjacent the player area presents images on adisplay screen that is positioned near the ball throwing device and infront of a batter. Video cameras are positioned in front of and onopposite sides of the anticipated trajectory of a hit baseball.

During use when a baseball delivered by the ball throwing device is hitby the batter and passes through the fields of the view of the videocameras, images of the baseball are captured and a streak showing thepath of the baseball through the fields of view is determined. Thestreak is used to simulate the flight of the baseball and to update theimage presented on the display screen thereby to simulate a battingexperience.

Although the above references disclose sports simulation systems thatcapture images of launched projectiles and use the image data tosimulate the flight of the launched projectiles, these sports simulationsystems fail to provide “true to life” sports experiences as a result ofthe mechanisms used to track the path of the launched projectiles.

Above-incorporated U.S. Patent Application Publication No.US2006/0063574 to Richardson et al. discloses a sports simulation systemcomprising a projectile tracking apparatus having a display surface onwhich a three-dimensional sports scene is presented. The projectiletracking apparatus captures images of a projectile tracking regiondisposed in front of the display surface to detect a launched projectiletraveling through the projectile tracking region towards the displaysurface. At least one processing stage communicates with the projectiletracking apparatus and is responsive to the data received from theprojectile tracking apparatus to determine the three-dimensionalpositions, velocity, acceleration and spin of a detected projectiletraveling through the projectile tracking region. The determinedthree-dimensional positions, velocity, acceleration and spin are used bythe at least one processing stage to calculate a trajectory of thelaunched projectile into the three-dimensional sports scene. Updatedimage data is generated by the at least one processing stage thatincludes a simulation of the launched projectile into thethree-dimensional sports scene following the calculated trajectory. Aprojection unit coupled to the at least one processing stage receivesthe image data from the at least one processing stage and presents thethree-dimensional sports scene, including the simulation, on the displaysurface.

Although this sports simulation system provides a better and morerealistic sports experience, in certain environments such as forexample, during club fitting, teaching and swing analysis, more accuratesimulations are desired. It is therefore an object of the presentinvention to provide a novel sports simulation system and a novelprojectile tracking apparatus.

SUMMARY OF THE INVENTION

Accordingly in one aspect there is provided a sports simulation systemcomprising a projectile tracking apparatus comprising at least twoimaging devices capturing images of a projectile tracking regiondisposed in front of a display surface from different vantages to detecta launched projectile traveling through said projectile tracking regiontowards said display surface; a projectile spin sensing unit capturingimages of a region at least partially overlapping with said projectiletracking region, each captured image comprising a projectile trailrepresenting a travel path of said projectile when a projectile ispresent in said region during image capture; and at least one processingstage receiving data from the imaging devices and said projectile spinsensing unit and determining the three-dimensional positions, velocity,acceleration and spin of a detected launched projectile travelingthrough said projectile tracking region, the three-dimensionalpositions, velocity, acceleration and spin being used by said at leastone processing stage to calculate a trajectory of said launchedprojectile into a three-dimensional sports scene.

According to another aspect there is provided a sports simulation systemcomprising a projectile tracking apparatus including a frameencompassing a display surface on which a video sequence portraying athree-dimensional sports scene is presented; and at least one pair ofdigital camera devices mounted on said frame and having fields of viewlooking across and in front of said display surface that overlap andencompass a projectile tracking region, each of said digital cameradevices including a first processor for processing image data andgenerating two-dimensional projectile coordinates when a projectiletravels through said projectile tracking region and is captured inimages acquired by said digital camera devices; a projectile spinsensing unit capturing images of a region at least partially overlappingwith said projectile tracking region, each captured image comprising aprojectile trail representing a travel path of said projectile when aprojectile is present in said region during image capture; a hostprocessor communicating with said digital camera devices and saidprojectile spin sensing unit, said host processor calculating athree-dimensional trajectory of said projectile taking into accountprojectile spin using the two-dimensional projectile coordinatesreceived from each first processor and the image data output of saidlaunch area sensing unit and outputting image data including saidcalculated three-dimensional trajectory; and a display unit receivingsaid image data and presenting said video sequence including asimulation of said calculated trajectory on said display surface.

According to yet another aspect there is provided a sports simulationsystem comprising: at least one pair of imaging devices havingoverlapping fields of view looking across and in front of a displaysurface from different vantages; a projectile spin sensing unitcapturing images of a region in front of said display surface, eachcaptured image comprising a projectile trail representing a travel pathof said projectile when a projectile is present in said region duringimage capture; at least one processing stage processing image data fromthe imaging devices and from said projectile spin sensing unit anddetermining the three-dimensional positions, velocity, acceleration andspin of a detected launched projectile traveling through saidoverlapping fields of view, the three-dimensional positions, velocity,acceleration and spin being used by said at least one processing stageto calculate a trajectory of said launched projectile into athree-dimensional sports scene projected onto said display surface; anda projection unit presenting said three-dimensional sport scene on saiddisplay surface including a simulation of said projectile following saidcalculated trajectory.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments will now be described more fully with reference to theaccompanying drawings in which:

FIG. 1 is a perspective of a sports simulation system;

FIG. 2 is a side elevation view of the sports simulation system of FIG.1;

FIG. 3 is a top plan view of the sports simulation system of FIG. 1;

FIG. 4 is a front elevation view of a projectile tracking apparatusforming part of the sports simulation system of FIG. 1;

FIG. 5 is an enlarged front elevation view, partly in section, of aportion of the projectile tracking apparatus of FIG. 4;

FIG. 6 is a side schematic view of a projectile launch area sensing unitforming part of the sports simulation system of FIG. 1;

FIG. 7 is a schematic perspective view of a projectile spin sensing unitforming part of the sports simulation system of FIG. 1;

FIG. 8 is a schematic block diagram of an area-scan digital cameraforming part of the projectile spin sensing unit of FIG. 7;

FIG. 9 is a schematic block diagram of an illumination board driver andillumination boards forming part of the projectile spin sensing unit ofFIG. 7;

FIG. 10 shows a backward spinning launched golf ball;

FIGS. 11 to 13 b are flowcharts showing steps performed during playerinteraction with the sports simulation system of FIG. 1;

FIG. 14 is an overhead view of a golf club making impact with a golfball within a launch area of the sports simulation system of FIG. 1; and

FIG. 15 shows processing of captured images to determine golf ball spinand golf ball spin tilt axis.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Turning now to FIG. 1, a sports simulation system is shown and isgenerally identified by reference numeral 100. As can be seen, sportssimulation system 100 includes a projectile tracking apparatus 102disposed in front of a projectile launch or hitting area A in which aplayer P stands. In this embodiment, the separation distance between thelaunch area A and the projectile tracking apparatus is approximately ten(10) feet. An overhead projectile launch area sensing unit 103 isdisposed above the launch area A. An overhead projectile spin sensingunit 105 is positioned between the launch area A and the projectiletracking apparatus 102. A host computer 104 is coupled to the projectiletracking apparatus 102, the projectile launch area sensing unit 103 andthe projectile spin sensing unit 105 via a high-speed serial data linkand to a ceiling mounted front video projector 106 that is aimed at theprojectile tracking apparatus 102. The host computer 104 outputs videoimage data to the projector 106, which in turn projects a video sequenceon the projectile tracking apparatus 102. The video sequence portrays athree-dimensional sports scene including a target at which a projectileis to be launched. In this embodiment, the sports simulation system 100simulates golf and thus, the three-dimensional sports scene is golfrelated and comprises an image of a golf course hole, practice rangeetc. The projectile to be launched at the projectile tracking apparatus102 of course is a golf ball GB.

The projectile tracking apparatus 102 outputs two-dimensional projectileposition data to the host computer 104 when the launched golf ball GBtravels through a projectile tracking region monitored by the projectiletracking apparatus. The projectile launch area sensing unit 103 outputsimage data representing the motion of the golf club through the launcharea A before, during and after impact with the golf ball to hostcomputer 104. The projectile spin sensing unit 105 outputs image data tothe host computer 104 that allows the host computer to determine thespin and the spin tilt axis of the golf ball GB as the golf ball travelsthrough the projectile tracking region. The host computer 104 in turnprocesses the two-dimensional projectile position data, the projectilelaunch area sensing unit image data and the projectile spin sensing unitimage data to determine the three-dimensional positions, launchvelocity, acceleration, side spin, backspin, spin tilt axis and launchangle of the golf ball so that the trajectory of the golf ball can beaccurately calculated. The calculated trajectory is then used todetermine a sports result and to update the image data conveyed to theprojector 106 so that the presented video sequence shows a simulation ofthe golf ball travel into the three-dimensional scene as well as thedetermined sports result.

FIGS. 2 to 5 better illustrate the projectile tracking apparatus 102. Ascan be seen, the projectile tracking apparatus 102 comprises an upright,inverted U-shaped frame 110 having a pair of side posts 112 and acrossbar 114 extending between the upper ends of the posts 112. A screen122 is supported by the frame 110. In this embodiment, the screen 122has a 4:3 aspect ratio making it particularly suited for displayingconventional television images. Those of skill in the art will however,appreciate that other image formats can be used. The screen 122 isloosely fastened to the back of the frame 110 at spaced locations.

The screen 122 includes multiple layers and is designed to reduceprojectile bounce as well as enhance protection behind the screen. Thefirst or front layer of the screen 122 is formed of highly reflectivenylon having some elasticity to resist permanent stretching/pocketingand abrasion. As a result, the front layer provides an excellent displaysurface 124 on which images projected by the projector 106 arepresented. The second or intermediate layer of the screen 122 is formedof soft and thick material and is designed to absorb projectile energywith reduced elastic effect thereby to inhibit stretching and or damageto the front layer. The third or back layer of the screen 122 is formedof a tough heavy canvas to which the intermediate layer can transferenergy. The back layer also inhibits excess deformation of theintermediate layer when contacted by a launched projectile. As a result,if the projectile tracking apparatus 102 is placed adjacent a wallsurface or the like, the back layer protects the surface behind thescreen 122 from projectile strike thereby to inhibit damage to thesurface and/or significant projectile rebound. If a space is providedbehind the projectile tracking apparatus 102, the back layer providesample protection for the space.

Imaging devices, in this embodiment a pair of high speed digital cameras128, are accommodated within the frame 110 with each camera beingpositioned adjacent a different top corner of the frame. Thus, thedigital cameras 128 are positioned in front of the player P and to theleft side and right side of the anticipated projectile path. The digitalcameras 128 are also angled to point downwardly and towards the playerposition so that the fields of view of the digital cameras are generallyperpendicular and overlap in the projectile tracking region whichextends from the projectile launch point to the screen 122. In thismanner, the path of the projectile can be tracked generally continuouslyfrom its launch point until it impacts the screen 122 and then as itrebounds from the screen 122.

In this embodiment, each digital camera 128 has at least a 640 by 480pixel array and includes built-in processing capabilities comprisingfield programmable gate arrays, a high performance 32-bit microprocessorand high speed memory. The distributed processing capabilities achievedby using the digital cameras 128 and the host computer 104 allow thedigital cameras to be operated at very high frame rates thereby allowingmultiple images of a fast moving projectile to be captured as theprojectile travels through the projectile tracking region 120. This isdue to the fact that the digital cameras 128 need only send data to thehost computer 104 relating to images in which projectile motion has beendetected allowing high speed projectiles to be tracked without excessivebandwidth between the host computer 104 and the digital cameras 128being needed. For example, in the case of a projectile travellingthrough the projectile tracking region 120 at a speed of 200 miles perhour, the frame rates of the digital cameras 128 are selected such thatat least four images of the projectile are captured by each digitalcamera 128. The viewing angles of the digital cameras 128 and thedimensions of the frame 110 are selected to provide the digital cameras128 with a resolving accuracy of approximately 1 mm per pixel. As aresult, a small projectile such as a golf ball will activateapproximately 12 pixels per image. This resolving accuracy enables evensmall, very fast moving launched projectiles to be readily determined incaptured images and as a result, reduces false projectile detection.

The on-board microprocessor of each digital camera 128 executes a motiondetection routine to determine if a projectile exists in the capturedimages and if so, whether the projectile satisfies specified motiondetection parameters defining a projectile characteristic signature. Theprojectile characteristic signature is used to ensure the detectedprojectile has characteristics matching the projectile in question, inthis case, a struck golf ball. The projectile can therefore bedistinguished from other objects captured in the images such as forexample, the golf club head. In this example, the projectilecharacteristic signature specifies allowable projectile size, shape,reflectivity and speed.

Infrared (IR) light emitting diode (LED) arrays (not shown) are alsopositioned within the posts 112 beside the digital cameras 128. Theillumination axes of the IR LED arrays are generally coincident with theoptical axes OA of the digital cameras. Each IR LED array emits IRradiation that is directed into the projectile tracking region 120. Asthe digital cameras 128 are responsive to both visible and infraredlight, providing the background IR illumination allows the projectiletracking apparatus 102 to work well in a variety of ambient lightingconditions. In situations where a small fast moving projectile islaunched, the IR illumination allows for detection of the projectilewithout interfering with the visual quality of the displayed imagepresented on the screen 122.

Audio speakers 140 are provided on the posts 112 and are aimed forwardlytoward the launch area A. The audio speakers 140 are driven by an audioamplifier (not shown) accommodated within the frame 110. The audioamplifier receives audio input from the host computer 104 during playthat is conveyed to the audio speakers 140 for broadcast thereby toenhance the sports experience.

The projectile launch area sensing unit 103 is disposed directly overthe launch area A and comprises an area-scan digital camera 160, anangled mirror 162, a plurality of illuminators 164 in the form ofhalogen spotlights and a power supply (not shown) for the spotlights 164as shown in FIG. 6. The spotlights 164 are aimed to provide sufficientillumination in the launch area A to permit image capture withoutadversely affecting visibility of the image projected on the screen 122.The area-scan digital camera 160 is ceiling mounted in a horizontalorientation approximately ten (10) feet above the launch area A. Theoptical axis of the digital camera 160 is generally in line with thecenter of the mirror 162 so that the field of view of the area-scandigital camera 160 is re-directed downwardly and centered over thelaunch area A. In this embodiment, the field of view of the area-scandigital camera 160 encompasses a three (3) foot by three (3) footregion.

Similar to the digital cameras 128 in the projectile tracking apparatus102, the area-scan digital camera 160 comprises an on-board processorthat executes a motion detection routine. During execution of the motiondetection routine, as images are captured by the area-scan digitalcamera 160, the images are examined to determine if one or more movingobjects exist therein that satisfy specified motion parameters. In thisexample, the motion parameters are selected to allow the on-boardprocessor of the area-scan digital camera 160 to detect when either amoving golf club or moving golf ball or both is in captured images.Captured images including one or more moving objects satisfying thespecified motion parameters are sent to the host computer 104 forfurther processing.

The projectile spin sensing unit 105 comprises a ceiling mounted,horizontally oriented area-scan digital camera 170, an angled mirror172, a plurality of infrared (IR) illuminator boards 174 and a driver176 for the illuminator boards 174 as shown in FIG. 7. The optical axisof the digital camera 170 is generally in line with the center of themirror 172 so that the field of view of the digital camera 170 isre-directed and centered over a region that at least partially overlapswith the projectile tracking region. In this embodiment, the regionextends from the front of the launch area A towards the projectiletracking apparatus 102 and encompasses a three (3) foot by six (6) footregion.

FIG. 8 better illustrates the area-scan digital camera 170. In thisembodiment, the digital camera 170 comprises a CMOS image sensor 180having a 640 by 480 pixel array and a pixel size equal to about 9.9microns. The image sensor 180 looks through a lens 182 having a focusdistance of about twelve (12) millimeters. Such a lens has been found toprovide good area coverage while maintaining sufficient resolution. Thedigital camera 170 includes built-in processing capabilities comprisinga field programmable gate array (FPGA) 184, a high performancemicroprocessor 186 and a high speed memory buffer 188.

In this embodiment, the projectile spin sensing unit 105 comprises four(4) illuminator boards 174, with each illuminator board comprising anarray of light emitting diodes (LEDs). The illuminator boards 174 arearranged in a manner so that the region within the field of view of thedigital camera 170 is generally evenly illuminated when the LEDs of theilluminator boards 174 are on. The driver 176 comprises a pulsegenerator that drives each of the illuminator boards 174 simultaneouslyso that the LEDs of the illuminator boards 174 turn on and off in unisonat regular intervals. In this embodiment, the LEDs of the illuminatorboards 174 remain in the on state for a 0.1 millisecond duration andremain in the off state for a 1 millisecond duration.

The projector 106 preferably has a resolution of at least 800×600, atleast 1200 ANSI Lumens brightness, a short throw lens, vertical‘keystone’ correction, and the capacity to accept digital RGB computervideo signals, and NTSC/PAL baseband television video signals.Projectors having this set of features include the Epson Powerlite 820P,the Toshiba TDP-DI-US, the InFocus LP650 and the Sanyo XP30 for example.

The host computer 104 is a general purpose computing device. In thisembodiment, host computer is an IBM compatible personal computerincluding an Intel Pentium® processor, at least 128 MB SDRAM, ahigh-speed hard drive, and a DVD player. The host computer 104 alsoincludes a display adapter assembly including a reconfigurable 32-bitvideo memory buffer partitioned into three separate buffers. One of thebuffers is used to store primary foreground image data representing oneor more independent foreground action elements if appropriate for thesports scene being displayed. A second of the buffers is used to storebackground image data and the third buffer is used to store projectiletrajectory image data. The display adapter assembly treats theforeground action, background and projectile trajectory image data asoverlay image planes that are combined seamlessly to generate the videoimage data that is output to the projector 106. The overlay image planesare non-destructive so that when a foreground action element and/orprojectile moves over an underlying image plane it is not necessary toredraw the underlying image plane. To reduce peak processingrequirements, the host computer 104 updates the background image dataless frequently than the foreground image data. The host computer 104provides the output video image data to the projector 106 on a videooutput channel. The host computer 104 receives external video feeds on atelevision/satellite/cable input channel, a video game input channel andan Internet input channel.

The host computer 104 is mounted within a protective enclosure (notshown) having external connectors to enable the host computer 104 to becoupled to the projector 106, the projectile tracking apparatus 102, theprojectile launch area sensing unit 103 and the projectile spin sensingunit 105. The enclosure also includes external connectors to allow thehost computer 104 to receive the television/satellite/cable, externalvideo game and Internet feeds. An interactive touch screen is alsoprovided on the enclosure to allow a player to interact with the hostcomputer 104.

A high speed digital serial interface, such as for example IEEE1394, isused for communications between the host computer 104, the projectiletracking apparatus 102, the projectile launch area sensing unit 103 andthe projectile spin sensing unit 105. Using this standard interfaceprovides a low cost, high performance solution while avoiding use ofexpensive analog frame grabbers. The interface also simplifies wiring asthe digital cameras 128 can be daisy-chained without loss of signalintegrity.

The host computer 104 executes sports simulation software stored in theSDRAM. In this example, the sports simulation software includes a golfsimulation module that requires a player to hit the golf ball GB at thescreen 122 of the projectile tracking apparatus 102 in response to thevideo sequence displayed on the screen 122.

To provide a realistic playing experience, a high resolution elevationmap of the golf course terrain is used. The course terrain elevation mapis constructed from a combination of two-dimensional images that includeoverhead satellite and/or aerial photographs used in conjunction withdigital photographs taken from ground level. Using photogrammetrytechniques, these orthogonal views are combined together. Using commonpoints in the images i.e. edges of sand hazards, trees etc., athree-dimensional model is synthesized without requiring referencetargets to be applied to the terrain of interest.

During training, practice or game play, the host computer 104 outputsvideo image data to the projector 106 causing the projector 106 toproject a video sequence portraying a three-dimensional sports scene onthe display surface 124 that includes a target at which the projectileis to be launched (see step 500 in FIG. 11). The host computer 104 alsoconditions the digital cameras 128 to capture a background image of theprojectile tracking region 120 devoid of a projectile (step 502) andthen scan the projectile tracking region to look for the presence of alaunched projectile at a very high frame rate (step 504). The player isthen prompted to launch the golf ball GB at the screen 122 (step 506).At this stage, the digital cameras 128, the area-scan digital camera 160and the area-scan digital cameral 170 are conditioned to capture andprocess images.

To facilitate detection of golf ball spin, an elongate reflective orretroreflective marker 190 is provided on the golf ball GB (see FIG.10). In this embodiment, the marker is a 45 mm by 5 mm piece ofreflective tape adhered or otherwise secured to the golf ball GB. Priorto launch, the golf ball GB is preferably oriented so that the longdimension of the reflective tape 190 is parallel to the width of thescreen 122. As a result, after launch and while the golf ball backspinsas it travels through the field of view of the area-scan digital camera170, when the driver 176 turns the LED arrays of the illuminator boards174 on, the reflective tape 190 is clearly visible to the area-scandigital camera 170 at intervals.

When the player launches the projectile at the projectile trackingapparatus 102 by striking the golf ball with a golf club and theprojectile enters the projectile tracking region 120, the projectileappears in the images captured by the digital cameras 128. Thus, thedigital cameras 128 generally synchronously capture a series of imagesof the projectile as it travels from its launch point through theprojectile tracking region 120 to its contact point with the screen 122and then as the projectile rebounds off of the screen (step 508). Thecaptured images are in turn processed by the on-board processors of thedigital cameras 128 to determine if the captured images include adetected projectile satisfying the projectile characteristic signature.

If the detected projectile satisfies the projectile characteristicsignature, the images are further processed to determine the center ofmass of the projectile in each image and its position in rectangularcoordinates (step 510). As a result, a series of two-dimensionalrectangular coordinates representing the two-dimensional positions ofthe projectile as it travels through the projectile tracking region 120relative to each digital camera 128 is generated. The two-dimensionalrectangular coordinates generated by the digital cameras 128 are in turnconveyed to the host computer 104.

The area-scan digital camera 160 of the projectile launch area sensingunit 103 captures and processes images to look for the existence of aswinging golf club passing through the launch area A and the launchedgolf ball exiting the launch area A. When a swinging golf club andlaunched golf ball are detected, the area-scan digital camera 160outputs the captured images to the host computer 104.

The area-scan digital camera 170 of the projectile spin sensing unit 105captures images at a frame rate equal to about 100 frames per second(fps) and processes consecutive images to determine if the differencebetween consecutive images exceeds a threshold signifying the existenceof an object in motion. When the difference between consecutive imagesexceeds the threshold, images are further processed to determine if theobject in motion resembles a golf ball. If the object in motionresembles a golf ball, the images are sent to the host computer 104 forfurther processing.

Upon receipt of the projectile coordinates from the projectile trackingapparatus 102, the host computer 104 calculates the positions of theprojectile's center of mass in three-dimensional space throughout itstravel through the projectile tracking region 120 including itscollision and rebound with the screen 122 using triangulation techniques(see step 520 in FIG. 12). With the position of the projectile inthree-dimensional space known during its travel through the projectiletracking region 120 and knowing the frame rates of the digital cameras128, the host computer 104 calculates the launch velocity of theprojectile and the velocity of the projectile over each image frame(step 522). The host computer 104 then compares each calculated velocitywith the previously calculated velocity to determine the acceleration ofthe projectile (step 524).

Upon receipt of the image data from the projectile launch area sensingunit 103, the host computer 104 analyzes the club head swing path 200(see FIG. 14) to determine where the club head hits the golf ball GB andto determine the initial golf ball trajectory or launch angle afterbeing hit. The host computer 104 also defines a club head motion vector202 as the tangent line along the club head swing path 200. Byestimating the initial golf ball trajectory, a golf ball motion vector206 is measured. Using this vector, a club face vector 208 can bedetermined as the line perpendicular to the tangent 210 of the club faceat the impact point of the golf ball and the club face. By comparing theclub head motion vector 202 and the club face vector 208, adetermination can be made as to whether the club face is open or closedupon impact with the golf ball. The degree to which the club head motionvector 202 is not parallel to the club face vector 208 at the point ofimpact determines the amount of side spin that the golf ball will have.This enables the host computer 104 to calculate the side spin of thegolf ball based on the angle of the club face at the point of contactwith the golf ball as well as on the impact and rebound angles of theprojectile with and from the screen 122 (also step 524).

Upon receipt of the images from the projectile spin sensing unit 105,the host computer 104 selects the first image (see step 600 in FIG. 13a) and analyses the image to determine if the image includes a golf balltrail 192 (step 602) as shown in FIG. 15. The golf ball trail 192appears in images due to the fact that velocity of the golf ball GBexceeds the frame rate of the digital camera 170. If the image does notinclude a golf ball trail, the image is discarded and the next image isselected at step 600. If the selected image includes a golf ball trail192, the golf ball trail in the image is located (step 604) and is thenexamined to determine if it is valid (step 606). In particular, thelength and width of the golf ball trail are compared with the thresholdranges. If the golf ball trail is not valid, the selected image isdiscarded and the next image is selected at step 600. If the golf balltrail 192 is validated at step 606, the image with the valid golf balltrail is designated for further processing (step 608) and the processreverts back to step 600 where the next image is selected.

Once all of the images from the projectile spin sensing unit 105 havebeen selected and processed, the images designated for furtherprocessing at step 608 are subjected to an image intensity profileanalysis (step 610 in FIG. 13 b) thereby to generate a combined profileof the golf ball trail over consecutive images as shown in FIG. 15. Thegolf ball trail length L_(c) per image is determined by the cross pointsof the combined profile (step 612). The images are subjected to adaptivethresholding to identify high intensity regions 196 in the imagescorresponding to the illuminated reflective tape 190 (step 614). A groupof high intensity regions 196 corresponding to the reflective tape 190appears in each image due to the golf ball spin and the pulsedillumination provided by the illuminator boards 174. The distancebetween the group of high intensity regions 196 in each consecutiveimage is then determined and is represented by L_(t) in FIG. 15 (step616). The time T_(p) taken for the golf ball GB to make a singlerevolution is expressed as:

$T_{p} = {\frac{L_{t}}{L_{c}} \cdot T_{f}}$

where T_(f) is the frame rate of the digital camera 170.

The time T_(p) is calculated for each consecutive image designated forfurther processing at step 608 and the average single rotation time forthe golf ball GB to make a signal revolution is determined (step 618).The average single rotation time is then converted into convenient unitssuch as for example rotations per minute (rpms).

The ball spin tilt axis is then estimated for each image using theorientation of the high intensity regions 196 in each group and therelative angle between the longitudinal axis of the high intensityregions 196 and the longitudinal axis of the golf ball trail 192. Theaverage ball spin tilt axis over the consecutive images designated forfurther processing at step 608 is then determined (step 620).

With the three-dimensional positions, launch velocity, acceleration,side spin, launch angle, backspin and spin tilt axis of the projectileknown, the host computer 104 extrapolates an accurate trajectory for theprojectile allowing a realistic simulation of curved and/or arcingprojectiles to be generated (step 526). The computed projectiletrajectory is then used to determine a sports result by computing theintersection of the calculated projectile trajectory with the displayedvideo image (step 528). With the projectile trajectory computed and thesports result determined, the host computer 104 updates the image datathat is conveyed to the projector 106 so that the video sequencedisplayed on the display surface 124 of the screen 122 shows thesimulated flight of the projectile and the sports result (step 530).

During video sequence display, when a simulation of the projectileflight is shown a graphical duplicate of the projectile is projectedonto the display surface 124 of the screen 122 that begins its flightfrom the impact point of the projectile with the screen 122. In thismanner, the projectile appears to continue its trajectory into the videoscene thereby to achieve a realistic video effect. The three-dimensionalscene is then updated in accordance with the sports result, allowinggame play or practice to continue.

Although the sports simulation system 100 has been described asincluding a ceiling mounted front projector 106 in combination with ascreen 122, those of skill in the art will appreciate that alternativeprojection devices may be used. For example, a rear video projector maybe used to project images onto the rear surface of the display screen122.

Those of skill in the art will appreciate that the projectile trackingapparatus 102 may include imaging devices at different locations to viewthe projectile tracking region and detect the existence of a launchedprojectile. Those of skill in the art will also appreciate that thenumber of processing stages may be increased or decreased as desired tohandle processing of the digital camera image data effectively inreal-time and provide a realistic projectile simulation.

If desired, the projectile launch area sensing unit 103 and theprojectile spin sensing unit 105 may include additional cameras. Theprojectile launch area sensing unit 103 and projectile spin sensing unit105 may include any number of illuminators or none at all if the ambientlight conditions are sufficient to provide for adequate image capture.Further, although the projectile launch area sensing unit 103 andprojectile spin sensing unit 105 are shown to include mirrors tore-direct the fields of view of the area-scan digital cameras 160 and170, those of skill in the art will appreciate that the area-scandigital cameras may be oriented to look directly at the regions ofinterest. The projectile launch area sensing unit 103 and projectilespin sensing unit 105 may also be positioned at any convenient location.

Although the use of retroreflective tape on the golf ball is described,alternative markers on the golf ball may be used such as for example,retroreflective paint, highly reflective tape, highly reflective paintetc.

While the sports simulation system is described as simulating golf, itwill be appreciated that the sports simulation system may be used tosimulate other sports where a projectile is launched. In such cases, theprojectile characteristic signatures are updated to enable launchedprojectiles to be accurately tracked.

Although embodiments have been described above with reference to thedrawings, those of skill in the art will appreciate that variations andmodifications may be made without departing from the spirit and scopethereof as defined by the appended claims.

What is claimed is:
 1. A golf simulator comprising: at least one pair ofimaging devices having overlapping fields of view and configured tocapture images of a launched, spinning golf ball travelling from alaunch area spaced in front of said display surface towards a displaysurface; a sensing unit configured to capture images of a region infront of said display surface adjacent said launch area, each capturedimage comprising high intensity regions resulting from one or moreilluminated discrete reflective markings on said launched, spinning golfball; at least one processing stage configured to process image datafrom the imaging devices and from said sensing unit and to determine thethree-dimensional positions, velocity, acceleration and spin of thelaunched, spinning golf ball traveling through said overlapping fieldsof view, the three-dimensional positions, velocity, acceleration andspin being used by said at least one processing stage to calculate atrajectory of said launched, spinning golf ball into a sports sceneprojected onto said display surface; and a projection unit configured topresent said three-dimensional sport scene on said display surfaceincluding a simulation of said launched, spinning golf ball followingsaid calculated trajectory.
 2. A golf simulator according to claim 1wherein said sensing unit comprises at least one imaging device having afield of view aimed downwardly toward said display surface.
 3. A golfsimulator according to claim 2 wherein said sensing unit furthercomprises at least one illuminator to illuminate the field of view ofsaid at least one imaging device at intervals.
 4. A golf simulatoraccording to claim 3 wherein said at least one illuminator comprises anarray of light sources.
 5. A golf simulator according to claim 4 whereinsaid light sources are light emitting diodes.
 6. A golf simulatoraccording to claim 4 comprising a plurality of illuminators at spacedlocations with said field of view.
 7. A golf simulator according toclaim 1 wherein during processing of image data from said spin sensingunit, said at least one processing stage is configured to generate aprofile of the golf ball trail over a plurality of images, determine thegolf ball trail length per image of said plurality of images, thresholdeach image to identify the high intensity regions corresponding to theelongate marking on said launched, spinning golf ball, determine thedistance between the high intensity regions in each image, calculate thespin rate and tilt of the launched, spinning golf ball using thedetermined distance, the golf ball trail length and the spin sensingunit image frame rate and estimate a spin tilt axis of the golf ballusing the orientation of the high intensity regions in each image andthe relative angle between the longitudinal axis of the high intensityregions and the longitudinal axis of the golf ball trail.
 8. A golfsimulation system comprising: at least two digital camera devices havingoverlapping fields of view; an overhead launch area sensing unitpositioned above and aimed generally straight down onto a launch regionpositioned a distance in front of a display surface in which contactwith a golf ball is made using a golf club and capturing images of saidlaunch region, said overhead launch area sensing unit comprising atleast one camera capturing images of said launch region and at least oneilluminator adjacent said at least one camera, said at least oneilluminator illuminating said launch region from above; at least oneprocessing stage processing image data received from the camera devicesand said overhead launch area sensing unit and determining thethree-dimensional positions, velocity, acceleration and spin of adetected launched golf ball traveling from its launch point to saiddisplay surface, the three-dimensional positions, velocity, accelerationand spin being used by said at least one processing stage to calculate atrajectory of said launched golf ball into a golf scene projected ontosaid display surface, wherein during determination of the spin, said atleast one processing stage processes the images captured by saidoverhead launch area sensing unit to determine the swing path of thegolf club before, during and after contact with said golf ball, thelaunch angle of the golf ball after contact with said golf club, and thedegree by which the head of the golf club is open or closed at the pointof impact with the golf ball; and a projection unit presenting said golfscene on said display surface including a simulation of said golf ballfollowing said calculated trajectory.
 9. A golf simulation systemaccording to claim 8 wherein said at least one illuminator comprises aplurality of laterally spaced illuminators.
 10. A golf simulation systemaccording to claim 9 wherein said at least one camera comprises a singlearea-scan camera.
 11. A golf simulation system according to claim 10wherein said area-scan camera processes captured images to determine ifone or more moving objects satisfying specified motion criteria are inthe captured images, and if so outputs the captured images to said atleast one processing stage.
 12. A golf simulation system according toclaim 10 wherein the optical axis of said area-scan camera is generallycoincident with the center of said launch region.
 13. A golf simulationsystem according to claim 12 wherein said area-scan camera is mounted ina generally horizontal orientation and wherein said launch area sensingunit further comprises a mirror to re-direct the field of view of saidarea-scan camera downwardly into said launch region.
 14. A golfsimulation system according to claim 8 wherein said at least oneprocessing stage compares a club head motion vector with a club facevector to detect the degree by which the head of the golf club is openor closed.
 15. A golf simulation system according to claim 14 whereinsaid at least one processing stage determines the tangent of the swingpath at the point of impact to calculate said club head motion vectorand determines a line perpendicular to the tangent of the face of thegolf club at the point of impact to calculate the club face vector. 16.A golf simulation system according to claim 8 wherein the illuminationprovided by said at least one illuminator is sufficient for imagecapture without negatively affecting visibility of the golf scenepresented on the display surface
 17. A golf simulator comprising: adisplay surface on which a golf scene is presented; a launch regionspaced a distance from said display surface to define a golf ball travelregion between said launch region and said display surface andconfigured to permit a user to launch a golf ball therefrom that travelsthrough said golf ball travel region and impacts said display surface; alaunch area sensing unit comprising: at least one camera mounted on saidhousing and capturing images of said launch region from generallydirectly above; and at least one illuminator mounted on said housingadjacent said at least one camera, said at least one illuminatorilluminating said launch region from above; and at least one processorcommunicating with the launch area sensing unit and modifying the golfscene presented on said display surface in accordance with thetrajectory of the golf ball launched at said display surface.
 18. A golfsimulator according to claim 17 wherein said at least one illuminatorcomprises a plurality of laterally spaced illuminators mounted on saidhousing.
 19. A golf simulator according to claim 18 wherein said atleast one camera comprises a single area-scan camera.
 20. A golfsimulator according to claim 19 wherein said area-scan camera processescaptured images to determine if one or more moving objects satisfyingspecified motion criteria are in the captured images, and if so outputsthe captured images to said at least one processor.
 21. A golf simulatoraccording to claim 19 wherein the optical axis of said area-scan camerais generally coincident with the center of said launch region.
 22. Agolf simulator according to claim 19 wherein said area-scan camera ismounted in a generally horizontal orientation and wherein said launcharea sensing unit further comprises a mirror to re-direct the field ofview of said area-scan camera downwardly into said launch region.
 23. Agolf simulator according to claim 17 wherein said launch area sensingunit comprises a housing configured to be mounted generally directlyabove said launch region.